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Objective: 


The  objective  is  to  produce  a  hyperspectral  sensing  system  which  is  both  substantially  faster  and  more  discriminating  than 
any  currently  proposed  or  available. 

Approach: 

The  approach  envisions  an  integration  of  modern  wavelet  signal  processing  and  arrays  of  micromechanical/optoelectronic 
devices.  The  wavelets  processing  engine  would  provide  the  system  with  the  ability  to  filter  (both  temporally  and  spatially) 
in  almost  real  time  so  that  the  data  burden  currently  hobbling  the  hyperspectal  sensors  (a  cube  of  raw  data  which  is 
interrogated  after  collection)  can  be  avoided.  The  filtering  (a  version  of  high  dimensional  inner  products)  will  communicate 
with  the  devices  (in  one  realization  with  an  array  of  micro  mirrors)  so  that  digestible  displays  are  produced.  Earlier,  one 
dimensional  work  by  the  PI,  upgraded  the  acquisition  radar  for  the  Longbow  missile  system. 

Progress: 

Year:  2004  Month: 

PROGRESS  REPORT:  F49620-00-C-0040  FROM:  15  AUG  02  TO  14  AUG  03 

In  the  last  year,  Plain  Sight  Systems  (P55)  has  concentrated  efforts  on  moving  existing  designs  into  the  InfraRed  region, 
but  has  also  pursued  algorithm  implementation  and  Graphical  User  Interface  (GUI)  software,  Digital  Mirror  Device  (DMD) 
driving  refinements  including  new  pseudo-random-Walsh  based  multiplexing  modalities,  and  a  genera!  effort  to  field  the 
prototype  that  was  described  in  previous  reports  moving  the  prototype  out  of  the  lab  and  into  other  indoor  and  outdoor 
locations  for  data  collection  efforts.  The  Infra  Red  effort  has  involved  design  and  specification  of  a  system  incorporating 
almost  all  common  off-the-shelf  (COTS)  components,  as  well  as  another  improved  design  incorporating  custom  optical 
components  specifically  designed  for  Near  InfraRed  (NIR)  operation,  and  with  the  specifics  of  the  present  system  in  mind. 
Plain  Sight  Systems  has  procured  most  of  the  components  for  these  systems,  and  is  in  the  process  of  refining  the  design, 
and  finalizing  the  last  few  components.  This  effort  is  expected  to  lead  to  the  construction  of  prototypes  over  the  next  half 
to  ?  of  a  year. 

Year:  2005  Month:  02 

ANNUAL  REPORT  FOR:  F49620-00-C-0040 

In  the  last  year,  Plain  Sight  Systems  have  concentrated  efforts  on  building,  testing,  and  refining  designs  from  the  previous 
year  for  a  system  operating  in  the  Near  Infrared  (NIR)  region.  We  have  also  continued  to  pursue  algorithm  implementation 
and  graphical  user  interface(GUI)  software  and  Digital  Micro-mirror  Device  (DMD)driving  refinements  including  newer 
pseudo-random  wavelet  based  multiplexing  modalities.  As  a  result,  we  were  able  to  field  a  prototype  that  was  capable  of 
being  tripod  mounted  and  along  with  a  standard  computer,  could  be  moved  out  of  the  lab  and  into  other  indoor  and 
outdoor  locations  for  a  successful  realistic  field  data  collection  efforts. 
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Progress: 

Year:  2006  Month:  02  Final 

Final  report  for  F49620-00-C-0040 

In  the  last  year  (no-cost  extension),  Plain  Sight  Systems  reached  the  goal  of  successfully  building  its  second  NIR  standoff 
hyperspectral  imaging  system,  NSTIS,  the  Near-Infrared  Spectral  Target  Identification  System.  Data  collection  and 
development  of  associated  analysis  tools  took  place,  and  progress  was  made  in  the  use  of  NSTIS  as  an  integrated 
sensing  and  processing  platform  inc  collaboration  with  our  initial  customer  (Lockheed  Martin).  In  concert  with  development 
of  the  NSTIS  systems,  Plain  Sight  Systens  continued  to  develop  spectrally  tunable  MOEMS  based  light  sources  and 
associated  algorithms  for  direct  spectral  feature  measurements  via  collaboration  with  Yale  University  and  Kansas  State 
University.  These  were  and  continue  to  be  applied  in  real  time  chemical  imaging  tasks  such  as  diagnostic  pathology 
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Project  Goals 

The  overall  goal  of  the  project  was  to  demonstrate  the  feasibility  of  portable  MOEMS  based 
hyperspectral  imaging  systems,  and  to  provide  real  time  detection  and  chemical  surveillance 
capabilities,  by  tuning  the  data  collection  process  to  the  desired  chemical  information. 

Approach 

We  have  pursued,  through  feasibility  and  prototype  construction  phases, 


•  A  digital  mirror  array  (DMA)  based  still  hyperspectral  processing  camera 
permitting  direct  extraction  of  spatio-spectral  features  from  a  scene  of  interest. 

•  Mirror  modulating  algorithms  to  enable  video  rate  detection. 

•  Other  spatial  light  modulators  capable  of  processing  on  the  sensor,  including 
grating  light  modulators,  arrays  of  optical  switches,  and  others  capable  of 
operating  in  various  infrared  wavelength  regions. 

•  Digitally  controlled  optical  sensors. 


Accomplished  Milestones 

By  the  beginning  of  this  last  contract  year.  Plain  Sight  reached  the  goal  of  successfully  building 
its  first  N1R  standoff  hyperspectral  imaging  system,  NSTIS,  the  Near-Infrared  Spectral  Target 
Identification  System.  In  concert  with  development  of  the  first  NSTIS,  spectrally  tunable 
MOEMS  based  light  sources  and  associated  algorithms  for  direct  spectral  feature  measurements 
were  developed  via  collaboration  with  Yale  University  and  Kansas  State  University.  These  were 
and  continue  to  be  applied  in  real  time  chemical  imaging  tasks  such  as  diagnostic  pathology. 

During  the  no-cost  extension,  as  part  of  this  contract  and  in  conjunction  with  cost  sharing  and 
other  funding  sources,  Plain  Sight  assembled  the  second  NSTIS,  performed  and  supported  data 
collection  and  development  of  associated  analysis  tools,  and  made  progress  in  the  use  of  NSTIS 


as  an  integrated  sensing  and  processing  platform  in  collaboration  with  our  initial  customer 
(Lockheed  Martin). 

Future  Milestones 
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•  Continued  testing  with  Lockheed  and  also  a  new  customer  -  the  Marine  Battle  Labs 

•  Optimization  of  NSTIS  and  movement  into  the  mid-infrared  with  other  SLMs 

•  Portable  MEMS  based  hyperspectral  system  capable  of  video  rate  chemometry 

•  Optical  switch  synaptic  systems  for  optical  data  processing  on  the  sensor 


Demonstration  Activity 

•  Coded  aperture  spectrograph. 

•  Spatio/spectral  target  detection. 

•  Spectral  tuned  light  generation. 

•  Tunable  broadband  filtering. 

•  Multifunctional  spectrometry. 

•  Direct  chemometric  modes. 


Remarks 

Our  goal  was  to  have  a  prototype  portable  demonstration  field  device  capable  of  a  variety  of 
hyperspectral  image  acquisition  modes  when  fitted  with  appropriate  optical  attachments.  This 
goal  has  been  achieved  for  the  visible  and  near  infrared  wavelength  regions.  We  have  also 
finished  the  development  of  a  second  and  improved  near  infrared  version  of  the  initial  DMA 
based  system  that  demonstrates  improved  image  quality.  Furthermore,  we  are  now  in  a  position 
to  propose  and  build  a  third  generation  system  capable  of  demonstrating  1)  higher  optical 
performance  and  throughput,  2)  faster  frame  rates,  3)  greater  portability,  and  4)  without  loss  of 
performance,  the  ability  to  simultaneously  analyze  two  wavelength  regions  or  to  supply  a  second 
channel  for  broadband  imaging  for  the  purpose  of  targeting  and  motion  tracking. 


Technical  approach  and  resource  allocation  on  the  project 

In  this  final  (no-cost  extension)  phase,  the  effort  concentrated  on  data  collection,  hardware 
drivers,  design  and  prototyping  of  instrumentation  around  the  DMA  by  Plain  Sight  Systems.  No 
work  was  performed  by  subcontractors  during  this  phase. 
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We  have  constructed  and  tested  novel  prototype  passive  hyperspectral  imaging  systems  based  on 
the  modality  of  replacing  the  slit  of  an  imaging  spectrograph  with  a  Digital  Micromirror  Array  or 
Device  (DMA  or  DMD).  The  advantages  of  this  novel  configuration  include  the  ability  to  scan 
with  the  DM  A,  rather  than  by  macro-mechanical  motion  of  the  camera  relative  to  the  target  of 
interest,  and  the  ability  to  collect  data  in  an  adaptively  multiplexed  way. 

In  the  final  2  years,  Plain  Sight  Systems  concentrated  efforts  on  building,  testing,  and  refining 
designs  for  a  system  operating  in  the  Near  Infrared  (NIR)  region.  We  have  also  continued  to 
pursue  algorithm  implementation  and  GUI  software,  and  DMD  driving  refinements  including 
newer  pseudo-random  wavelet  based  multiplexing  modalities.  As  a  result,  we  were  able  to  field 
two  prototypes  operating  in  the  NIR  region  that  were  capable  of  being  tripod  mounted  and,  along 
with  a  standard  computer,  could  be  moved  out  of  the  lab  and  into  other  indoor  and  outdoor 
locations  for  successful  realistic  field  data  collection  efforts.  One  of  these  systems  was 
successfully  tested  at  a  military  facility  (China  Lake  NAVAIR)  and  was  subsequently  delivered 
to  a  contractor  (Lockheed  Martin)  for  further  evaluation  and  development.  The  second  system  is 
to  be  scheduled  for  demonstration  at  another  military  facility  (Marine  Battle  Labs)  during  2006. 

The  fielding  of  the  first  NIR  system  originally  involved  the  design  and  specification  of  a  system 
incorporating  almost  all  COTS  components  with  the  exception  of  some  that  were  recoated  for  the 
NIR  spectral  range.  However,  replacing  some  of  the  COTS  design  with  an  optical  configuration 
invented  in  house  outside  the  scope  of  this  contract,  we  were  able  to  overcome  difficulties  of  the 
NIR  region  and  some  of  the  major  design  challenges  of  using  the  DMA  in  an  imaging  system. 
Also,  by  analyzing  collected  data  and  refining  the  existing  design,  we  were  able  to  gain  valuable 
experience  that  has  led  to  the  refinement  of  the  design  and  the  build  of  a  similar  second  NIR 
system. 


Optical  Concept 

This  spectral  imaging  system  is  based  on  a  dispersive  imaging  spectrograph  layout  as  shown  in 
Figure  1 .  In  conventional  imaging  spectrographs,  the  entrance  aperture  is  a  vertical  rectangular 
slit.  Typically  one  must  translate  the  image  across  the  width  of  the  slit.  We  have  replaced  the 
typical  entrance  slit  with  a  DMD  aperture  where  the  DMD  is  also  at  the  focal  plane  of  an  F- 
mount  (Nikon)  or  FD-mount  (Canon)  camera  lens.  The  target  “object”  is  focused  onto  the  DMD 
and  is  scanned  into  the  imaging  spectrograph  system  using  programmable  aperture  encoding 
methods.  In  conventional  raster  scanning  modality,  a  rectangular  subset  of  DMD  mirrors  that  is 
orthogonal  to  the  dispersive  dimension  of  the  imaging  spectrograph  is  selected  to  sequentially 
illuminate  the  subsequent  optical  system.  This  effects  a  spatial  scanning  of  a  single  entrance  slit 
allowing  that  portion  of  the  image  on  the  DMD  to  enter  the  optical  system.  The  spatial 
dislocation  in  the  dispersion  axis  of  the  optics  affects  an  angular  difference  in  the  incident  rays 
upon  the  grating.  This  results  in  a  translation  of  the  spatio-spectral  image  at  the  focal  plane  of  the 
array  detector.  Scanning  the  image  can  be  accomplished  using  Hadamard  or  a  combination  of 
Hadamard  and  Raster  scanning. 
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Figure  1:  Dispersive  imaging  spectrograph  layout  and  modifications:  In  the  conventional 
imaging  spectrograph,  an  incoming  image  (1)  of  a  scene  to  be  analyzed  (2),  is  focused,  by  a  lens 
(3),  onto  a  plane  mask  which  contains  a  single  horizontal  slit  (4).  In  this  way,  a  one-dimensional 
horizontal  slice  (5)  through  the  original  image  (1)  passes  through  the  mask  (4).  A  dispersion 
device  (6)  disperses  vertically  the  slice-image  (5).  The  result  is  that  each  point  in  the  slice-image 
(5)  is  dispersed  into  a  vertical  spectrum  of  the  light  at  that  point.  This  results  in  a  2D 
spectrograph  image  (7).  The  spectrograph  image  (7)  is  re-imaged  by  optics  (not  shown)  onto  a 
detector  array  (8).  A  hyper-spectral  image  of  a  scene  is  obtained  by  then  scanning,  one  height  at 
a  time,  across  the  scene  (e.g.  from  top  to  bottom).  This  is  typically  done  by  actually  moving  the 
apparatus  with  respect  to  the  scene,  or,  alternatively,  with  a  scanning  mirror. 

In  the  modified  configuration,  a  spatial  light  modulator  (9),  such  as  a  TI  DMD  is  used  in  place  of 
the  mask  (4).  In  this  way,  the  multiplex  advantage  is  available  to  the  system.  Also,  scanning  can 
be  accomplished  electronically,  without  the  need  for  relative  motion  of  the  device  with  respect  to 
the  scene,  nor  the  need  for  macro-moving  parts. 
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Development  Timeline 

1st  system  (with  imSpector,  on  optical  bench)  (project  start  ->  9/2001) 

This  compact  system  demonstrated  replacement  of  mechanical  scanning  with  DMD  scanning, 
and  provided  early  demonstrations  of  the  multiplex  advantage.  System  artifacts  were  severe,  and 
the  system  alignment  unstable.  The  device  was  not  usable  beyond  these  simple  demonstrations 
that  the  concept  had  merit. 

This  early  system  was  based  on  an  active  light  design,  in  which  the  DMD  is  illuminated  with 
broadband  light,  which  is  then  re-imaged  through  the  sample,  into  a  modified  imaging 
spectrograph.  The  spectrograph  modification  consists  of  removing  the  entrance  slit  and  placing 
the  sample  to  sit  at  the  plane  formerly  occupied  by  the  slit.  There  were  actually  2  distinct 
embodiments  of  this  setup.  Figure  2  shows  one  of  them. 


2nd  System  (with  imSpector™  and  prism,  on  optical  bench)  (9/2001  ->  12/2001) 

In  this  iteration,  a  TIR  prism  assembly  was  used  to  fold  the  optical  path,  enabling  an  optical 
benchtop  prototype  that  was  stable  enough  to  collect  data  over  a  period  of  time.  This  system  was 
used  to  demonstrate  the  adaptive  multiplex  advantage,  and  demonstrated  the  feasibility  and  merit 
of  making  a  portable  prototype.  Figure  3  shows  this  system. 
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Figure  3:  Photo  of  2nd  early  system,  and  some  data  collected  while  imaging  a  plant  leaf. 


3rd  System  (with  grating  and  genl)  (1/2002  ->  7/2002) 

This  iteration  was  intended  to  be  a  portable  prototype  based  on  the  previous  system,  but  the 
imSpector™  was  replaced  by  a  grating,  in  order  to  allow  flexible  choice  of  gratings  for  selection 
of  wavelength  ranges,  and  to  better  characterize  the  dependence  of  system  performance  on 
component  choices.  The  system  is  shown  uncovered  in  Figure  4. 


Figure  4:  Photos  of  3rd  system  uncovered. 
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4th  Generation:  Fielding  the  Portable  Visible  Integrated  System  (7/2002  ->  9/2003) 

In  this  iteration,  Plain  Sight  concentrated  on  imaging  scenes  with  natural  or  typical  light  such  as 
sunlight,  room  light,  and  other  standard  electrical  lighting.  Other  iterations  had  used  laboratory- 
controlled  lighting.  Also,  this  year  we  characterized  the  efficacy  of  our  system  in  a  portable 
mode.  This  amounted  to  placing  the  now-covered  camera  housing  on  a  rolling  cart,  together 
with  a  PC  equipped  with  a  flat-panel  display  for  compactness,  and  a  long  extension  cord. 

We  also  updated  the  system  software  to  allow  reliable  acquisitions  at  15  to  20  frames  per  second, 
where  before  this  had  been  a  peak  rate  that  was  not  reliably  achieved.  This  helped  us  to  mitigate 
the  effects  of  moving  cloud  cover  and  other  typical  phenomena  associated  with  outdoor  imaging. 
Another  line  of  development  involved  customized  pseudo-random  multiplex  codes.  These  codes 
allowed  us  to  mitigate  a  number  of  artifacts  that  had  been  reported  in  the  previous  years’ 
developments.  We  were  able  to  collect  quality  datacubes  from  ordinary  objects  using  a  variety  of 
indoor  and  outdoor  lighting  conditions. 


Figure  5:  Views  of  4th  generation  system  mounted  on  rolling  cart. 
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Figure  6:  Various  spectra  and  pseudo-colored  images  of  data  collected  by  the  system. 
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Near  Infrared  System  (12/2002  ->  Present) 

In  the  past  year.  Plain  Sight  successfully  built  its  first  NIR  hyperspectral  camera  prototype, 
NST1S,  the  Near-Infrared  Spectral  Target  Identification  System.  It  operates  by  the  same 
fundamental  principles  as  our  previous  visible  wavelength  range  device,  SLM-based 
multiplexing  with  no  macro-moving  parts,  but  its  design  was  enhanced  to  allow  it  to  operate  in 
the  900nm  to  1700nm  region. 


Figure  7 :  Several  views,  electronics  exposed,  of  the  NSTIS  camera.  The  leftmost  shows  the 
system  from  the  front;  center  view  from  the  back;  rightmost  from  the  side. 

NSTIS  Specifications: 


Spectral  Sampling 

1.5625nm 

Spectral  Resolution 

<10nm 

Working  F# 

<5.6 

Typical  Data  Collection  Time 

1 0  sec  /  datacube 

Datacube  Dimensions 

up  to  512x512x532 

NSTIS  Components: 

•  Housing  -  The  NSTIS  housing  consists  of  custom  made,  black-painted  aluminum  parts. 
The  fully  assembled  system  is  a  non-rectangular  shape  that  fits  roughly  into  a  size  of  24“ 
wideX  16”  High  x  21”  Deep. 

•  Optics  -  NSTIS  contains  custom-mounted  optical  components  designed  for  operation  at 
F#’s  no  less  than  F5.6.  These  are  delicate,  precision  aligned  components  and  are  not  user 
serviceable.  The  internal  custom  relay  lenses  meet  the  proposed  target  design 
specifications:  Object  diagonal  of  12.309mm  (tilted  object  creates  a  reduced  extent  for 
lens  design),  Min.  working  distance  of  36mm,  Mag  1.47,  MTF  15  lp/mm  at  40%,  F# 

2.44,  Distortion  less  than  0.5%,  Transmission  greater  than  80%  over  operational 
bandwidth,  Vignetting  less  than  5%,  Wavelength  range  900nm  -  1700nm,  AR  coating  R 
less  than  1 .5%  over  spectral  range  and  at  marginal  ray  angles,  Chromatic  aberrations  less 
than  5  microns,  Minimum  back  focal  length:  100  mm,  “Telecentric  like”  performance  in 
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object  space.  The  system  fore  optic  (mounted  on  the  front  of  the  system)  is  a  Canon- 
mounted  lens  system.  Any  standard  Canon  manual  FD-mount  lens  may  be  used. 

•  DMD  &  DMD  Controller  -  NSTIS  contains  an  848X600  TI  DLP  chip  (DMD)  and  a 
custom  controller  electronics  controller  board. 

•  Camera  -  NSTIS  includes  an  Indigo  Phoenix™  large-area  InGaAs  camera,  with  RTIE 
electronics. 

•  Software  &  System  Controller  -  NSTIS  is  controlled  via  a  Windows  XP  based 
computer  (HP  workstation  xw4100  or  equivalent),  and  includes  a  National  instruments 
PCI-1422  Digital  frame  grabber,  and  a  Cyber  Research  CYINT  32P  controller  board. 
The  system  is  fully  configured  for  use,  and  should  require  no  user  service. 

•  Power  Requirements  -  1 20V  60Hz  AC  power,  rated  for  at  least  20  Amps. 


The  key  attributes  of  this  NIR  hyperspectral  imaging  system  are  its  modular  design  based  on 
COTS  parts  with  no  macro-moving  parts,  multiplexed  data  collection  for  increased  sensitivity 
and  frame  rates,  and  the  capability  of  adaptable  measurement  schemes  which  make  the  prototype 
a  promising  platform  for  integrated  sensor  processing  (ISP). 

The  following  figures  illustrate  and  describe  the  basic  optical  path  of  the  system. 


Figure  8:  NSTIS  optical  path  and  layout  with  ray  trace. 
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Figure  9:  Solid  model  of  NSTIS  optical  path  and  layout  with  labels  which  describe: 


1 .  Camera  lens  front  objective: 

a.  Standard  Canon  manual  FD 
mount 

b.  Modular 

c.  Transfers  image  of  scene  into 
system 

2.  All-reflective  relay  system: 

a.  Transfers  image  from  objective, 
conditioning  it  for  DMD 

b.  Custom  PSS  Optical  design 

c.  Flat  to  tilted  image  plane 
correction 

d.  COTS  spherical  reflectors 

3.  TIR  Prism: 

a.  Transfers  image  onto  and  off  of 
DMD 

b.  Folds  optical  path 

c.  Custom  PSS  design  and  coating 
for  NIR  wavelength  region 

4.  DMD: 

a.  Multiplexing  component 

b.  Modulates  system  resolution 
elements 

c.  COTS  TI  MEMS  mirror  array 

d.  PSS  custom  control  electronics 

5.  Diffraction  Grating: 

a.  Dispersion  component 

b.  Spectrally  spreads  each  “pixel” 
from  the  DMD 

6.  NIR  Broadband  camera: 

a.  Indigo  Phoenix  NIR  camera 

b.  900nm-  1700nm 

c.  640  x  512  pixels 
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Because  of  the  tight  placement  of  the  DMD  at  the  center  of  the  optical  path,  previous  control 
electronics  would  not  fit  within  the  system  illustrated  above.  As  a  result,  it  was  necessary  to 
design  and  fabricate  a  new  flexible  circuit  board,  mated  to  our  existing  electronics,  that  allows 
for  better  positioning  of  the  DMD  and  greater  access  to  the  component  for  optical  purposes. 
These  components  are  shown  the  next  figure. 


Figure  10:  DMD  driving  electronics  and  flex  circuit  board  (not  to  scale). 


System  Software: 

NSTIS  is  operated  with  the  real-time  user  interface  application  NstisAppR,  a  GUI  that  supports 
hyperspectral  image  preview,  acquisition,  review,  and  storage.  It  handles  all  communication  and 
control  between  the  PC,  the  DMD  electronics,  and  the  Indigo  Phoenix™  camera.  NstisAppR  is 
primarily  a  data  collection  application  with  a  few  simple  analysis  tools  included.  More 
substantial  post-collection  data  processing  capabilities  may  be  added  in  future  versions. 


The  exact  type  of  acquisition  as  well  as  any  post-acquisition  processing  is  determined  by  settings 
in  the  parameter  dialog  as  shown  in  the  next  figure. 
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Figure  12:  NstisAppR  parameters. 


NSTIS  supports  the  two  scan  modes,  “multiplexed”  (Hadamard)  and  “raster”.  In  multiplexed 
scan  mode,  multiple  overlapping  spectra  from  various  columns  of  the  image  are  acquired  per 
frame  (multiplexed)  and  then  mathematically  transformed  (de-multiplexed)  to  reconstruct  the 
hyperspectral  datacube.  In  raster  scan  mode,  only  the  spectra  corresponding  to  a  single  column  of 
the  image  are  acquired  per  frame.  Multiplex  mode  enables  data  acquisition  in  lower  light 
situations,  while  raster  scan  is  appropriate  when  there  is  too  much  light  for  multiplexing. 

NSTIS  supports  image  acquisition  at  normal  and  high-resolution.  In  the  optical  configuration  of 
NSTIS,  each  pixel  of  the  Phoenix  camera  “sees”  4  pixels  of  the  DLP.  When  normal  resolution  is 
selected,  the  4  pixels  are  averaged.  In  high-resolution  acquisition  mode,  the  4  pixels  are  used  to 
sub-modulate  the  pixels  of  the  camera,  thus  enhancing  system  resolution.  When  high-resolution 
mode  is  used,  the  result  is  a  datacube  that  is  8  X  larger  (2  X  in  each  dimension),  enhancing 
datacube  size  from  256x256x266  to  512x512x532. 


After  acquiring  a  data  cube  NstisAppR  can  immediately  perform  two  post-acquisition 
procedures:  an  inverse  transform  procedure  when  in  multiplex  mode  and  a  deskew  procedure. 

When  the  Deskew  check  box  is  checked,  the  acquired  data  will  be  deskewed.  This  means  that  the 
spatio-spectral  direction  of  the  system  will  be  “unwound”,  so  that  the  datacube  will  be  presented 
as  2  purely  spatial  dimensions,  and  1  purely  spectral  dimension. 

Acquired  data  cubes  can  be  viewed,  wavelength  by  wavelength,  in  an  automated  or  user- 
controlled  way  via  a  slider  control  as  show  in  the  next  figure. 
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Figure  13:  Reviewing  an  acquired  cube. 


Acquired  data  cubes  can  be  saved  into  disk  files  and  later  reloaded  into  NstisAppR.  Cubes  can 
also  be  exported  to  file  for  further  processing  in  Hyperspectral  Explorer,  HSE,  optionally 
available  with  NSTIS,  as  well  as  in  MATLAB  .  Data  can  also  be  exported  to  a  variety  of  other 
file  formats  as  well  as  directly  into  the  HSE  application. 
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Sample  Data: 

The  following  figures  show  hyperspectral  images  and  sample  spectra  as  acquired  using  the 
NSTIS  system. 


Figure  14:  A  scene  consisting  of  a 
number  of  different  plastics  and 
other  materials  imaged  with  the 
system  and  displayed  at  one 
wavelength.  Spectra  of  the  various 
plastics  in  the  scene,  as  acquired  by 
the  device,  are  shown  below  the 
image  as  indicated.  Some  analysis  of 
this  hyperspectral  dataset  using  HSE 
is  demonstrated  in  the  next  figure. 


ft  II 
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Figure  15:  This  figure  shows  screenshots  of  analysis  performed  on  the  hyperspectral  dataset  of 
the  previous  figure  using  the  HyperSpectral  Explorer  platform  (HSE).  The  top  image  shows  the 
user-selected  regions  of  the  image  designated  as  training  sets.  The  subsequent  screenshots  each 
show  the  resulting  factor  analyses.  Materials  of  different  chemical  compositions  become 
differentiated. 
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Figure  16:  Spectrum  of  Ethanol  measured  by  imaging  a  gas  cell.  Broadband  image  on  left; 
extracted  raw  spectrum  on  right. 


Figure  17:  Spectrum  of  Propane  measured  by  imaging  a  gas  cell.  Broadband  image  on  left; 
extracted  raw  spectrum  on  right. 
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Figure  18:  Spectrum  of  palms  (real  and  artificial)  in  NIR  hyperspectral  image  collected  indoors. 


Figure  19:  Screenshot  of  NstisAppR  with  acquired  image  of  polystyrene  ruler  illustrating  the 
basic  analysis  tools  included;  advanced  processing  on  the  data  is  to  be  done  elsewhere  with  other 
software  (e.g.  HSE). 
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Clear  liquids  In  test  tubes  and  a  bottle 


50  100  150  200  250 


Figure  20:  The  top  image  is  of  a  set  of  vials  containing  various  transparent  chemicals.  It  has 
been  pseudo-colored  to  reflect  the  chemistry  revealed  by  the  underlying  hyperspectral  data. 
Whereas  the  chemicals  are  transparent  and  identical  in  appearance  in  visible  light  to  the  human 
eye,  this  shows  them  with  different  tints  of  color  reflecting  their  different  chemistries  as 
measured  in  the  NIR.  The  lower  image  shows  the  raw  spectra  measured  by  the  camera  from 
each  of  the  chemicals  in  the  various  containers. 
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Second  Build  of  Near  Infrared  System  (8/2004  ->  Present) 

In  the  past  year,  Plain  Sight  successfully  completed  and  tested  a  second  NIR  hyperspectral 
camera  prototype  (NSTIS2),  incorporating  design  changes  and  improvements  yielded  from 
experience  gained  in  building  and  fielding  the  first  completed  system.  The  following  images 
show  the  system  at  various  stages  of  assembly. 


Figure  21 :  Main  optical  bench  completed  and  displayed  with  uncoated  optical  mounting 
components. 


Figure  22:  Fully  assembled  system  with  covers  removed  to  display  internal  optics  and 
electronics. 
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Figure  23:  NSTIS2 


Figure  23:  Completed  NST1S2  system  deployed  with  PC  in  a  mobile  configuration.  The  system 
is  shown  imaging  a  natural  scene  through  the  window  of  building. 


Data  Example  from  NSTIS2 

In  the  following  example,  a  transparent  panel  of  Lucite  and  portion  of  camouflage  tarp  were 
hidden  in  a  natural  scene.  As  illustrated  by  Figure  24,  an  image  of  the  scene  taken  by  a  regular 
RGB  digital  camera,  these  objects  were  difficult  to  pick  out  with  the  naked  eye. 
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Portion  of 
camouflage  fabric 
wrapped  around 
tree  trunk 


Panel  of  Lucite 
leaning  against 
tree  trunk 


Figure  24;  Natural  scene  imaged  by  conventional  high  resolution  RGB  digital  camera  with 
hidden  items  indicated. 


Figure  25:  Selected  feature  image  from  hyperspectral  datacube  of  scene  acquired  by  NSTIS2 
system.  Both  the  camouflage  fabric  and  Lucite  panel  appear  dark  with  respect  to  background 
and  are  no  longer  difficult  to  see. 
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China  Lake  Data  Collection  and  Analysis  -  Demonstration  of  Chemical  Sensing 
(9/2004  ->  Present) 

Work  was  performed  under  this  contract  in  support  of  field  studies  and  development  described  in 
this  section  in  so  far  as  it  was  related  to  hardware  and  design  modifications.  Other  support  and 
activity  by  Plain  Sight  Systems  and  other  parties  involved,  was  chiefly  funded  by  other  sources 
and  contracts  as  well  as  Plain  Sight  cost-share.  We  include  documentation  of  this  activity  as  it 
was  in  part  enabled  by  our  work  under  this  contract  and  also  to  illustrate  the  success  of  the 
NSTIS  system  in  demonstrating  the  technology  developed  under  this  contract. 

In  September  2004,  the  NSTIS  system  was  sent  to  the  China  Lake  Naval  facility  where,  in 
collaboration  with  NAVAIR  and  a  group  from  Lockheed  Martin’s  Missiles  and  Fire  Control 
division.  Plain  Sight  collected  a  significant  amount  of  data  of  various  targets  of  interest.  This 
activity  was  also  partially  funded  by  other  sources  as  well  as  Plain  Sight  cost-share.  Prior  to 
deployment,  work  was  done  hardening  the  mounts,  modifying  the  base  to  mate  to  a  specific 
tripod,  improving  ESD  protection,  and  evaluating  a  series  of  imaging  optics  both  custom  and 
COTS. 


Figure  26:  NSTIS  camera  fielded  at  NAVAIR  China  Lake  facility.  The  system  was  fielded  in  a 
mobile  configuration  with  tripod  and  telescopic  imaging  lens.  Majority  of  data  was  collected 
through  an  open  window  on  the  upper  floor  of  a  large  building. 


23 


-24- 


Figure  27:  RGB  digital  camera  image  of  view  from  vantage  point  of  NSTIS  camera  location. 

In  the  scene  of  Figure  27,  we  see  a  camouflaged  truck  that  was  almost  a  quarter  mile  distance 
from  the  location  of  the  NSTIS  camera.  In  front  of  the  truck  is  a  person  next  to  whose  feet  were 
placed  glass  containers  that  were  imaged  filled  with  various  chemicals.  The  glass  and  chemicals 
were  all  transparent  and  not  visible  to  the  naked  eye  at  that  distance.  The  next  image  shows  data 
acquired  by  NSTIS,  from  the  same  vantage  point,  with  a  telescopic  lens  focused  on  the  region  of 
interest. 
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Figure  28:  A  selected  narrow  NIR  wavelength  band  image  of  datacube  acquired  by  NST1S  with 
telescopic  lens.  Labels  indicate  the  location  of  the  two  chemical  containers  and  the  back  of  the 
truck  that  was  also  shown  in  Figure  27. 


As  we  see  in  Figure  28,  the  chemicals  that  were  not  visible  to  the  naked  eye  now  begin  to 
become  visible  in  a  selected  narrow  NIR  wavelength  band  in  which  they  both  absorb  energy.  The 
following  figures  illustrate  further  processing  of  the  data  and  resulting  spectral  features  that 
easily  distinguish  one  gas  from  the  other. 
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Figure  29  :  Sample  spectra  for  each  chemical  extracted  from  the  hyperspectral  datacube. 


Figure  30  :  Sample  analysis  using  Hyperspectral  Explorer  finds  spectral  features  which  locate 
and  discriminate  one  chemical  from  the  other. 


Figure  31  :  Using  this  analysis  of  spectral  features,  a  pseudo-colored  image  of  the  region  of 
interest  is  produced  that  both  locates  and  discriminates  the  two  chemicals. 
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Lockheed  Martin  Data  Collection  and  Analysis  (4/2005  ->  Present) 

Work  was  performed  under  this  contract  in  support  of  field  studies  and  development  described  in 
this  section  in  so  far  as  it  was  related  to  hardware  and  design  modifications.  Other  support  and 
activity  by  Plain  Sight  Systems  and  other  parties  involved,  was  chiefly  funded  by  other  sources 
and  contracts  as  well  as  Plain  Sight  cost-share.  We  include  documentation  of  this  activity  as  it 
was  in  part  enabled  by  our  work  under  this  contract  and  also  to  illustrate  the  success  of  the 
NSTIS  system  in  demonstrating  the  technology  developed  under  this  contract. 

The  main  data  for  the  study  was  collected  at  the  Lockheed  Martin  Orlando  facility  in  April  2005. 
From  a  collection  tower,  a  set  of  datacubes  was  collected  for  a  background  scene  as  well  as  for  a 
target  (a  vehicle)  moving  right  to  left  at  various  time  steps  (see  Figures  32  and  33).  The  data  was 
used  to  demonstrate  detection  and  localization  capabilities  of  an  integrated  sensing  and 
processing  suite  consisting  of  dynamically  programmable  DMA  sensor  (NSTIS)  and  adaptive 
algorithms. 


Figure  32:  NIR  broadband  image  of  sample  datacube  acquired  by  NSTIS  of  parking  lot,  roads, 
and  buildings  as  viewed  from  vantage  point  at  Lockheed  Martin  Orlando  facility. 


Figure  33:  Image  from  localized  datacube  acquired  by  NSTIS  with  target  box  indicated  in  red 
that  was  used  to  train  algorithm. 

Some  results  of  this  activity  were  described  the  in  the  article:  “An  Application  of  Integrated 
Sensing  and  Processing  Decision  Trees  for  Target  Detection  and  Localization  on  Digital  Mirror 
Array  Imagery”  by  Priebe,  Marchette,  Park,  and  Muise,  submitted  for  publication  in  Applied 
Optics.  A  draft  of  this  article  is  included  as  an  appendix  to  this  document  to  illustrate  a 
demonstration  of  integrated  sensing  and  processing  using  NSTIS  in  the  field. 
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Future  Plans 

Plain  Sight  is  enthusiastic  to  continue  to  purse  and  develop  this  technology  and  the  possible 
opportunity  to  insert  it  into  defense  applications  as  well  as  some  in  the  commercial  sector. 
Together  with  a  group  from  Lockheed  Martin’s  Missiles  and  Fire  Control  division,  we  plan  to 
continue  a  series  of  tests  and  demonstrations  with  various  possible  customers  from  defense 
agencies.  It  is  already  the  case  that  the  NSTIS  system  is  to  be  scheduled  for  demonstration  at 
another  military  facility  (Marine  Battle  Labs)  during  2006. 

Also  currently  under  investigation  are  possible  applications  to  the  environmental  and 
petrochemical  industries  that  Plain  Sight  has  been  developing  through  work  outside  the  scope  of 
this  contract. 
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Concepts  Demonstrated 

Scanning  without  macro-motion 

Prior  art  passive  hyperspectral  imaging  systems  have  all  required  macro-moving  parts  to  achieve 
hyperspectral  imaging  of  a  two  dimensional  scene:  either  physical  scanning  of  the  camera,  the 
object  of  interest,  or  some  macro-mechanical  part  such  as  a  rotating  prism  or  a  scanning  mirror 
were  required.  By  scanning  through  a  sequence  of  slits  or  encodements  on  the  DMD,  the  present 
system  can  image  a  full  hyperspectral  datacube  without  any  such  motion  (only  MEMS  array 
elements  move,  by  about  4  microns  maximum  displacement  at  the  comers  of  each  element).  Of 
course,  hyperspectral  imaging  can  also  be  achieved  with  tunable  filters  that  effectively  raster 
scan  in  spectral  domain.  Although  such  systems  are  simple,  they  are  inefficient  and  do  not  have 
multiplexing  capabilities. 

Multiplex  advantage 

The  following  sequence  of  images  demonstrates  the  multiplex  advantage  associated  with 
Hadamard  encoded  scans  versus  raster  scans.  The  experimental  setup  consisted  of  the  third 
prototype  system  described  above  with  a  Sigma  Lens  (50mm,  f2.8.  Macro  Lens,  Nikon  mount), 
focused  for  roughly  one  to  one  imaging  on  a  sample  of  didymium,  scotch-taped  to  a  sheet  of  20# 
bond  copy  paper,  with  a  blotch  of  ordinary  ballpoint  pen  ink  on  the  scotch-tape.  The  scene  was 
illuminated  from  behind  with  a  LabSphere™  integrating  sphere  with  an  IHLS-1 00-125  125  W 
Halogen  light  bulb.  The  resulting  amount  of  light  was  somewhere  between  that  present  in  a 
scene  illuminated  by  bright  sunlight  and  that  of  one  illuminated  by  typical  indoor  office  lighting. 


SNR  for  Hadamard  vs.  Raster  image  acquisition 
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Figure  34:  SNR  graph  demonstrating  multiplex  advantage  (3rd  generation  system). 
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Figure35:  Image  sequence  demonstrating 
multiplex  advantage. 


-31  - 


Adaptive  multiplex  advantage 

The  preceding  section  demonstrated  that  multiplexing  is  an  advantage  in  low  light  situations,  and 
actually  causes  artifacts  when  there  is  plenty  of  light.  Hence,  depending  on  the  amount  of  light 
present  in  a  scene,  it  might  not  always  be  optimal  to  multiplex.  Moreover,  one  can  raster  scan 
through  a  tiling  of  sub-regions  in  a  scene,  and  multiplex  within  each  sub-region,  in  an  adaptive 
fashion.  In  this  way,  the  device  can  be  used  to  automatically  multiplex  at  an  optimal  level 
(provided  that  lighting  conditions  do  not  change  much  over  the  time  scale  of  a  pair  of  scans). 

The  following  graph  demonstrates  the  added  advantage  achieved  by  multiplexing  at  the 
appropriate  scale. 


SNR  vs.  shutter  speed  for  raster,  Walsh,  and  best  level 


Figure  36:  SNR  graph  demonstrating  adaptive  multiplex  advantage  (3rd  generation  device). 


Feasibility  of  video-rate  passive  hyperspectral  image  data  acquisition 

The  data  above  demonstrate  that  in  appropriate  lighting  conditions,  our  device  can  collect  raw 
video  frames  at  about  9000fps.  Since  a  typical  datacube  is  640x480x128,  this  amounts  to 
collection  of  about  70  full  hyperspectral  frames  per  second.  Modem  DSPs  are  roughly  capable 
of  providing  relevant  processing  of  the  resulting  ~2.5Gigabytes/sec  of  data,  if  highly  optimized, 
and  offline  preprocessing  is  allowed.  Compression  and  adaptive  measurement  schemes  being 
investigated  under  this  project  could  enable  more  sophisticated  processed  acquisitions  with  less 
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processing  power.  While  the  data  shown  above  reflects  electronic  shutter  speed  and  not  actual 
frame  rate,  experiments  performed  with  a  RedLake™  high-speed  CMOS  imaging  system 
demonstrated  the  feasibility  of  actual  high-speed  image  acquisition  at  several  thousand  raw 
frames  per  second  with  our  current  system. 

Feasibility  of  natural  Light  Hyperspectral  Imaging  with  Multiplex  Advantage 

In  2003,  Plain  Sight  demonstrated  the  ability  of  the  system  to  collect  hyperspectral  datacubes  in 
ordinary  room  light,  and  in  sunlight.  As  mentioned  above,  we  were  able  to  collect  quality 
datacubes  from  ordinary  objects  using  ordinary  indoor  commercial  overhead  fluorescent  lighting, 
as  well  as  a  number  of  datacubes  with  the  device  pointed  out  the  window  of  our  facility, 
spectrally  imaging  automobiles,  trees  and  grass  in  the  area.  The  system’s  electronic  control  and 
shutter  allows  it  to  be  easily  tunable  to  different  conditions.  The  device  was  sensitive  enough  that 
we  were  able  to  image  outdoors  all  the  way  at  F/22.  We  were  able  to  collect  data  until  about 
7:45pm  Eastern  Time  in  summer,  when  the  sunlight  is  fading  and  dusk  is  settling  in. 

System  Performance  Characterization 

Spatial  resolution  -  This  is  determined  by  the  swappable  fore-optics.  The  system  is  typically 
used  in  a  one  to  one  imaging  mode,  so  that  individual  pixel  spatial  resolution  is  nominally  17 
microns,  and  line  pair  resolutions  at  about  twice  this  dimension  have  been  typical  with  the 
current  prototype.  When  mounted  on  a  microscope,  the  total  system  spatial  resolution  scales 
accordingly. 

Spectral  range  and  resolution 

Nominal  spectral  resolution  is  on  the  order  of  2nm,  depending  on  system  alignment,  and  could 
be  made  higher  or  lower  with  a  different  choice  of  grating.  The  spectral  range  is  roughly  400run 
-  1  lOOnm  or  900nm  -  1700nm,  with  approximately  350  pixels  of  data  per  cube,  in  the  spectral 
direction.  Typical  spectral  resolution,  FWHM,  is  about  15nm. 


Data  collection  speed 

The  system  has  collected  reasonable  data  at  shutter  speeds  as  fast  as  1/10000  sec.  At  the 
moment,  acquisition  and  processing  electronics  are  the  bottleneck  to  faster  data  acquisition,  in 
spite  of  the  fact  that  sub-optimal  opto-mechanical  components  are  employed  in  the  current 
system.  The  existing  prototype  collects  20  fps  in  its  fastest  mode,  for  a  rate  of  one 
640X480X128  datacube  every  6.4  sec.  With  a  fully  integrated  and  engineered  solution,  the 
system  could  collect  data  at  video  rates,  as  described  above. 

Light  power  at  image  sensor 

In  one  experiment,  the  light  power  at  an  image  sensor  was  measured  to  be: 

Background:  .011pW/cm2 

Raster  scan:  .124  pW/cm2 

Hadamard  Scan  (128  elements):  2.56  pW/cm2 
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Aberrations  and  other  problems  and  lessons 

Sensor  non-linearity  and  multiplexing  artifacts 

As  long  as  the  system  was  not  blooming  the  multiplexing  helped  to  achieve  low  light  level 
imaging.  When  the  sensor  bloomed,  artifacts  were  present.  The  first  few  multiplexed  images  in 
Figure  23  demonstrate  this  effect.  The  present  system  is  capable  of  adapting  to  varying  light 
conditions  as  described  previously. 

System  misalignment  and  multiplex  artifacts 

As  a  prototype  static  hyperspectral  imaging  system,  device  performance  and  design  goals  were 
achieved  and  even  surpassed.  However  the  system  is  prone  to  alignment  and  stability  issues  that 
could  be  solved  by  an  opto-mechanical  engineering  effort  and  funding  to  accomplish  this. 

Multiplexed  artifacts  are  predicted  and  observed  in  association  with  a  data  skew  resulting  from 
these  alignment  and  optical  design  issues.  It  is  not  currently  possible  to  have  the  system  both 
horizontally  and  vertically  aligned:  there  is  a  trapezoidal  image  plane.  Our  multiplex  algorithms 
assume  a  rectangular  acquisition.  Because  multiplexed  data  are  quantized  by  the  A/D  converter 
of  the  image  acquisition,  this  results  in  an  artifact  with  structure  related  to  the  multiplexing 
scheme  used.  Walsh  multiplexing  results  in  a  multi-scale  grid  artifact,  and  pseudo-random 
Hadamard  codes  can  be  used  to  mitigate  these  effects.  These  issues  are  present,  but  are  minor 
under  most  lighting  conditions  (i.e.  measurable  only  in  the  bottom  bit  of  the  A/D).  The  effect 
can  be  mitigated  by  pre-computation  that  would  dictate  a  modified  mirror  driving  setup  that 
compensates  for  the  geometric  aberrations. 

Vignetting 

Vignetting  is  severe  in  this  system  and  in  some  instances  possibly  in  excess  of  50%  due  to  some 
of  the  cots  optical  elements  employed  in  the  imaging  spectrograph  portion  of  the  camera  system. 
The  selection  of  better  optics  would  resolve  this  issue. 


Spectral  Target  and  Anomaly  Detection 

The  following  two  examples  demonstrate  target  and  anomaly  detection  enabled  by  the 
hyperspectral  imaging  system. 
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orthogonalized  vehicle  detection  filter. 
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Figure  38:  Fake  vegetation  is  discovered  easily  through  the  standoff  coded  aperture 
imager.  Bottom  right;  a  conventional  spectrograph  image  under  same  light  integration 
setting. 


Figure  39:  Camouflage  detection  simulation  images.  Selected  filter  designs  for 
discrimination.  The  left  image  represents  a  combination  of  bands  converted  to  RGB. 
The  right  image  uses  best  discrimination  spectral  filters  converted  to  RGB 
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OPTICAL  ISSUES 

ForeOptics:  The  DMD  is  placed  at  the  focal  plane  of  a  COTS  Nikon  optical  system  and  exhibits 
excellent  imaging  properties  and  has  a  vast  array  of  COTS  lensing  options.  However,  exact 
placement  of  the  DMD  at  the  focal  plane  has  not  been  realized  with  the  GEN1  prototype  and  TIR 
prism.  We  expect  improved  spatial  resolution  with  subsequent  generations.  We  have  the 
advantage  of  selecting  from  the  array  of  Nikon  imaging  optics  as  our  applications  are  identified 
in  more  detail.  The  latest  NSTIS  system  with  Canon  FD  foreoptics  has  already  greatly  improved 
upon  this. 

TIR  prism:  The  COTS  total  internal  reflection  (TIR)  prism  allows  a  more  compact  optical  path 
and  was  engineered  for  a  commercially  available  projector  system.  There  are  some  stray  light 
and  dispersion  issues  relating  to  the  TIR  prism  geometry  that  may  reduce  effectiveness  for  some 
low  light  applications. 

We  have  found  a  multiple-bounce  reflection  image  that  manifests  itself  as  an  intensity  modulated 
spatio-spectral  image  when  the  prism  and  grating  are  a  particular  coincident  optical 
configuration.  Testing  and  examination  of  this  phenomenon  can  be  accounted  for  and  reduced  or 
eliminated  by  rotating  the  relative  scanning/dispersion  dimension  by  90  degrees  to  the  TIR 
prism. 

Additional  improvements  can  be  made  by  optical  engineering  efforts  to  produce  a  TIR  prism 
assembly  that  is  adapted  to  this  application  that  will  allow 

•  Lower  F#  operation/improved  throughput 

•  Improved  spectral  resolution 

•  Reduced  stray  light 

•  Improved  optical  geometry 

•  More  compact  designs 

Imaging  Optics  -  Spectrograph:  To  keep  costs  to  a  minimum  we  have  employed  inexpensive 
COTS  imaging  optics  in  our  spectrograph  optics.  This  resulted  in  a  sacrifice  in  F#  and  FOV  that 
did  not  include  the  entire  DMD  object  plane.  Improving  the  selection  of  imaging  optics  in  the 
spectrograph  portion  of  the  system  will  improve  performance  dramatically. 

•  Reduced  vignetting 

•  Increase  throughput  (F#  reduction  to  match  NIKON-DMD  optics) 

•  Improved  spatial  resolution 

•  Improved  spectral  resolution  (Reduced  chromatic  aberrations  in  the  NIR) 

•  Improved  spectral  response  (VIS-NIR  broadband  AR  coatings) 

Sensor:  The  sensor  employed  for  our  first  proof  of  concept  device  was  based  on  a  standard  video 
camera,  low  performance  and  not  of  scientific  quality,  and  did  not  match  DMD  pixels  or  format. 
The  latest  NSTIS  system  improves  greatly  upon  this,  employing  a  high  quality  sensor  chosen  to 
mate  appropriately  with  the  designed  surrounding  optics. 
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1.  Appendix:  Related  publication  demonstrating  Integrated 
Sensor  Processing  (ISP)  work  done  with  NSTIS  system. 

Priebe  et  al.,  “An  Application  of  Integrated  Sensing  and  Processing  Decision  Trees  for  Target 
Detection  and  Localization  on  Digital  Mirror  Array  Imagery”,  submitted  for  publication  in 
Applied  Optics.  Draft  dated  July  18,  2005. 
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We  demonstrate  the  applicability  of  the  Integrated  Sensing  and  Processing 
Decision  Trees  (ISPDTs)  methodology  to  a  set  of  digital  mirror  array  (DMA) 
hyperspectral  imagery.  In  particular,  we  demonstrate  that  ISPDTs  can  be 
used  to  detect  and  localize  targets  using  just  a  few  DMA  Hadamard  frames, 
so  that  an  entire  hyperspectral  data  cube  need  not  be  collected  in  order  to 
successfully  perform  the  given  task.  This  suggests  that  such  an  integrated 
sensing/processing  suite  may  be  appropriate  for  extremely  time-sensitive 
pattern  recognition  applications. 

©  2005  Optical  Society  of  America 

OCIS  codes:  100.2960,  100.5010. 

1.  Introduction 

We  demonstrate  the  applicability  of  the  Integrated  Sensing  and  Processing  Decision  Trees1 
(ISPDTs)  iterative  denoising  methodology  to  a  set  of  digital  mirror  array2  (DMA)  imagery. 

The  sensor  sequentially  collects  256  Hadamard  frames  in  order  to  produce  a  hyperspec¬ 
tral  data  cube  that  measures  256x256  pixels  spatially  and  consists  of  266  spectral  bands. 
However,  the  sensor  is  (dynamically)  programmable;  given  a  pattern  recognition  task  of  in¬ 
terest  (detect ion/ localization /classification)  it  is  desirable  to  learn  a  sequence  of  just  a  few 
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Hadamard  frames  that  will  accomplish  the  task  sans  the  requirement  to  collect  (and  process) 
the  entire  data  cube. 

We  demonstrate  the  construction,  on  one  training  hyperspectral  data  cube,  of  an  iterative 
denoising  tree  that  allows  for  detection  and  localization  of  a  target  using  just  two  Hadamard 
frames;  the  entire  hyperspectral  data  cube  need  not  be  collected  in  order  to  successfully 
perform  the  task. 

This  proof-of- concept  result  suggests  that  an  integrated  sensing/processing  suite  consisting 
of  a  dynamically  programmable  DMA  sensor  and  ISPDT  processing  may  be  appropriate  for 
extremely  time-sensitive  pattern  recognition  applications. 

2.  Sensor  &  Data 

2.  A.  Sensor 

The  sensor  under  consideration  is  a  prototype  system  developed  by  PlainSight  Systems  which 
incorporates  a  DMA  device  in  order  to  realize  a  Hadamard  multiplexed  imaging  system. 

The  known  signal-to-noise  (SNR)  advantage  in  Hadamard  spectroscopy3  extended  to  imag¬ 
ing  systems4,5  allows  for  the  collection  of  a  hyperspectral  cube  of  data  with  more  efficient 
light  collection  than  standard  “pushbroom”  hyperspectral  imagers. 

The  PlainSight  sensor  is  a  spatial  light  modulator-based  multiplexing  hyperspectral  imag¬ 
ing  camera,  operable  in  the  near-infrared  spectral  range  of  about  900  nm  -  1700  nrri.  The 
system  uses  a  DMA  commercially  available  from  Texas  Instruments  for  projector  display 
applications.  The  DMA  contains  848  columns  and  600  rows  of  mirrors  and  measures  10.2 
mm  x  13.6  mm. 

When  the  scene  is  illuminated  on  the  DMA  device,  a  standard  raster  scan  could  be  im¬ 
plemented  by  turning  the  first  column  of  mirrors  ON,  sending  this  column  to  a  diffraction 
grating  which  results  in  a  spectral  representation  of  the  first,  spatial  column  of  the  scene 
being  illuminated  on  the  detector  array. 

If  one  opens  multiple  slits  in  the  DMA  array,  the  detector  array  will  be  presented  with  the 
superposition  of  many  columns  of  spectra.  This  system  would  have  the  advantage  of  optimal 
SNR  when  the  pattern  of  the  open  slits  form  a  Hadamard  pattern.3  Each  individual  frame 
at  the  detector  array  has  less  physical  meaning  than  in  the  pushbroorn  method,  but  when  all 
the  patterns  of  the  Hadamard  sequence  have  been  recorded,  the  full  hyperspectral  datacube 
is  recoverable. 

The  PlainSight  sensor  implements  the  process  wherein  the  detector  array  is  a  standard 
Indigo  Phoenix  large-area  InGaAs  camera  operating  in  the  near-infrared  wavelengths. 

During  standard  operation  of  the  system,  the  sensor  collects  512  frames.  Each  frame  is 
522  x  256  pixels  and  represents  spectra  vs  spatial  row.  The  512  frames  that  are  collected 
are  based  upon  256  Walsh  (0s  and  Is)  patterns.  Since  the  theory  of  optimal  SNR  is  based 
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upon  Hadamard  (Is  and  -Is)  patterns,  one  needs  to  collect  two  Walsh  patterns  to  generate 
a  single  Hadamard  pattern.  Thus,  the  512  collected  frames  represent  the  required  Walsh 
patterns  to  form  a  full  set  of  256  Hadamard  patterns.  Since  each  column  in  the  DMA  array 
will  hit  the  diffraction  grating  at  a  different  location,  the  spectra  will  hit  the  detector  array 
at  a  different  location  per  column.  We  describe  this  as  a  “skewness”  in  spectra  and  results 
in  the  522  pixels  in  the  spectral  dimension  in  order  to  represent  the  266  actual  spectral  bins. 
Of  course,  this  spatial/spectral  mixing  and  skewness  is  invertible  once  all  256  Hadamard 
patterns  have  been  collected.  The  resultant  hyperspectral  scene  is  dimension  256x256  with 
266  spectral  bands  from  900  nm  to  1700  nm. 

The  sensor  has  the  capability  to  utilize  adaptive  Hadamard  frame  collection  schemes, 
as  follows.  The  sensor  is  dynamically  programmable,  in  that  it  can  be  tasked  to  render 
any  Walsh  pattern,  at  any  scale,  and  at  any  location  on  the  DMA.  For  example,  standard 
operation  of  the  sensor  renders  512  Walsh  patterns  from  the  lowest  spatial  frequency  to  the 
highest,  at  a  resolution  of  256  mirrors.  In  adaptive  operation  of  the  sensor,  any  one  of  these 
patterns  can  be  rendered  on  the  DMA,  an  image  frame  is  captured,  some  data  processing  can 
occur  leading  to  a  decision  regarding  the  next  frame,  and  then  any  of  the  other  patterns  can 
be  rendered.  This  sensor  capability  allows  one  to  task  the  DMA  to  render  patterns  that  have 
been  algorithmically  calculated  to  be  “optimal”  given  the  data  that  was  previously  observed. 
(This  adaptive  sensor  operation  implies  that  the  actual  hyperspectral  cube  may  no  longer 
be  recoverable  from  the  frames  collected,  as  one  requires  the  full  256  Hadamard  patterns 
be  collected  in  order  to  invert  the  spatial/spectral  mixing  caused  by  the  multiplexing.  If 
the  desired  processing  does  not  require  the  inverse  operation  be  available,  then  it.  is  more 
efficient  to  not  collect  all  the  data.  We  collect  only  what  the  algorithm  needs  to  perform 
its  function.)  Thus,  the  dynamic  programmability  of  the  sensor  allows  one  the  flexibility  to 
sense  only  that  data  which  is  required  for  the  image  processing  task  at  hand. 

Our  task  for  integrating  sensing  and  processing  is  to  develop  a  scheme  to  collect  these 
Hadamard  frames  one  at  a  time,  and  make  inferences  about  the  current  hyperspectral  scene 

and  about  the  next  Hadamard  frame  to  be  collected  —  as  the  frames  are  being  sensed. 
Application  may  involve  target  detection/localization/classification  as  well  as  object  track¬ 
ing. 

2.B.  Data 

Data  were  collected  at  Lockheed  Martin  Orlando  in  April  2005.  From  a  collection  tower,  a 
full  set  of  256  Hadamard  frames  was  collected  for  a  background  scene  (7o)  as  well  as  for  a 
target  (a  vehicle)  moving  right  to  left  at  four  time  steps  (/£,  t  =  1,2,3, 4).  Figure  1  depicts 
one  Hadamard  frame  at  time  t  =  2. 

This  collection  of  five  hyperspectral  scenes  is  used  to  demonstrate  detection  and  local- 
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ization  capabilities  of  an  integrated  sensing/processing  suite  consisting  of  a  dynamically 
programmable  DMA  sensor  and  ISPDT  processing. 

3.  Methodology 

We  apply  the  ISPDT  iterative  denoising  methodology  to  the  hyperspectral  vehicle  imagery, 
It,  t  =  0, 1, 2, 3, 4,  described  above. 

We  use  the  hyperspectral  data  cube  collected  at  time  t  =  2,  I<±,  as  our  training  data;  Figure 
1  depicts  Hadamard  frame  110  at  time  t  =  2.  We  focus  on  64x210  image  swaths,  so  as  to 
obviate  issues  of  sensor  skewness  and  edge  effects;  Figure  2  depicts  the  swath  for  Hadamard 
frame  110  of  the  training  data  I2,  with  the  region  delineated  by  the  small  box  representing  the 
training  information  regarding  the  target  of  interest.  (No  additional  information  regarding 
vehicle/non- vehicle  pixels  within  the  target  box  is  utilized.) 

Training  proceeds  as  follows;  see  Figure  3. 

3.  A.  Step  1 

No  single  Hadamard  frame  provides  adequate  performance  in  segregating  target  from  non¬ 
target  pixels,  and  so  Hadamard  frame  110  is  selected  for  use  at  the  root  by  virtue  of  its 
performance  in  providing  the  most  clustering  clarity  —  a  new  Hadamard  frame  at  each 
node  is  determined  based  on  the  best  separation  between  the  target  box  and  the  non-target 
pixels  by  using  the  adjusted  Rand  index  criterion.6,7  Model  based  clustering8  is  employed, 
in  which  a  Bayesian  Information  Criterion  (BIC)  is  used  to  determine  the  complexity  and 
type  of  a  Gaussian  mixture  fit  to  the  data.  The  clusters  are  then  defined  in  terms  of  the 
posterior  likelihoods  of  the  individual  components.  The  Hadamard  frame  110  pixels  (gray 
scale  intensity)  cluster  into  three  clusters;  the  spatial  location  of  the  pixels  for  these  three 
clusters  are  represented  by  blue,  green  and  yellow  in  the  root  node  of  the  tree  depicted  in 
Figure  3. 

3.B.  Step  2 

In  building  an  ISPDT,  after  clustering  at  a  node,  each  cluster  is  processed  in  a  branch  of  the 
tree.  This  subsequent  processing  proceeds  in  an  analogous  fashion  to  that  of  the  root:  if  the 
pattern  recognition  task  at  hand  can  be  adequately  addressed  (for  the  data  falling  to  that 
branch)  then  tree  growth  (along  that  branch)  is  halted.  It  is  necessary,  of  course,  to  perform 
the  search  for  the  best  Hadamard  frame  once  again  in  each  branch,  conditionally  upon  the 
results  of  all  previous  clusterings. 

In  this  application,  the  left-most  (green)  and  right-most  (yellow)  clusters  at  the  root  of 
the  ISPDT  depicted  in  Figure  3  yield  essentially  pure  non-target  branches  and  no  further 
tree  growth  is  necessary  in  these  branches,  while  the  middle  (blue)  root  cluster  contains 
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many  of  the  target  pixels  as  well  as  a  significant  number  of  the  non-target  pixels.  This 
middle  cluster  must  be  processed  further.  In  this  middle  cluster,  we  find  that  Hadamard 
frame  151  provides  clustering  into  two  clusters;  furthermore,  one  of  these  clusters  (the 
left-most  node  at  level  three  of  the  ISPDT  depicted  in  Figure  3)  contains  a  significant 
number  of  target  pixels  and  no  non-target  pixels.  Ergo,  this  leaf  is  labeled  as  the  “target  leaf” . 

Notice  that,  as  was  mentioned  above,  no  information  regarding  vehicle/non- vehicle  pixels 
within  the  target  box  is  utilized.  Nevertheless,  the  “target  leaf”  (the  left-most  node  at  level 
three  of  the  ISPDT  depicted  in  Figure  3)  successfully  —  and  unsupervisedly,  with  respect  to 
the  pixels  within  the  target  box  makes  this  distinction. 

The  clustering  at  each  level  of  the  ISPDT  can  be  used  to  produce  a  partition  of  the  input 
space.  Thus  the  ISPDT  described  above  and  depicted  in  Figure  3  can  be  presented  as 

g(x)  =  I{x  6  Cifi  A  x  g  C-jp}  (l) 

where  Cij  represents  the  jth  partition  cell  at  level  i  of  the  tree  and  g  is  a  classifier  for  pixel 
x. 

Note  that  while  the  entire  hyperspectral  data  cube  was  required  for  the  training  data, 
Equation  1  allows  processing  for  detection/localization/classification  with  the  collection  of 
just  two  Hadamard  frames,  110  and  151,  as  opposed  to  all  256. 

The  ISPDT  depicted  in  Figure  3  is  quite  simple.  In  general,  for  more  complex  applications, 
there  may  be  more  elaborate  conditionality  for  instance,  depending  upon  what  is  sensed 
in  the  first  few  frames,  different  choices  may  be  required  for  subsequent  sensing.  Such  a  tree 
results  in  multiple  target  leaves  and  requires  dynamic  programmability  of  the  sensor. 

4.  Results 

We  present  successful  detection/localization  results  on  the  vehicle  imagery  test  data  /,, 
t  =  0, 1,3,4,  described  above,  using  the  ISPDT  trained  on  /2  and  depicted  in  Figure  3  and 
Equation  1. 

There  is  no  vehicle  present  at  time  t  —  0,  and  (see  Figure  4)  no  pixels  fall  into  the  “target 
leaf”.  This  implies  no  detection,  as  desired. 

For  t  =  1, 3, 4  there  is  a  vehicle  present,  and  (see  Figures  5,  6,  7)  in  each  case  the  existence 
of  pixels  falling  into  the  “target  leaf”  implies  detection,  as  desired.  Furthermore,  the  spatial 
location  of  these  pixels  indicates  that  the  detection  is  indeed  “on  target” . 

Remark:  The  fact  that  not  all  target  pixels  fall  to  the  “target  leaf”  is  inconsequential; 
the  goal  is  detection  and  localization,  and  in  each  test  case  pixels  on  target  are  identified 
(i  =  1,3,4)  or  no  pixels  are  identified  (t  =  0)  as  desired. 
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4-A.  Additional  Illustration 

Figure  8  depicts  a  more  elaborate  ISPDT,  requiring  non-trivia!  conditioning/adaptation, 
generated  for  /3.  In  this  tree,  as  before,  Hadamard  frame  110  is  used  at  the  root,  and  in 
the  middle  cluster  at  level  two  Hadamard  frame  151  provides  clustering  into  two  clusters. 
While  the  “target  leaf”  - —  the  left- most  node  at  level  three  —  provides  target  detection, 
there  are  a  significant  number  of  target  pixels  that  fall  to  the  right-most  node  at  level  three 
—  a  non-target  node. 

When  training  on  I2,  tree-building  stops  at  this  point.  If  training  on  I3  (presented  in  Fig¬ 
ure  8)  it  is  preferable  to  continue  the  tree-building,  using  Hadamard  frame  145  for  additional 
accuracy.  Thus  the  left-most  node  at  level  four  is  an  additional  “target  leaf”  for  this  illus¬ 
tration,  and  the  two  target  leaves  together  allow  more  target  pixels  to  be  identified  (while 
still  yielding  no  false  detections).  This  tree  would  be  realized  by  dynamically  programming 
the  sensor  to  collect  Hadamard  frame  145  only  for  cases  in  which  frames  110  and  151  leave 
detection  unresolved. 

The  scatter  plot  tree  for  the  illustrative  ISPDT  depicted  in  Figure  8  is  given  in  Figure 
9.  Note  that  the  scatter  plots  are  horizontal  location  on  the  x-axis  versus  pixel  value  on 
the  y-axis.  Thus  separability  of  the  red  (target  box)  pixels  and  black  pixels  is  envisioned 
by  projecting  onto  the  y-axis.  For  example,  initial  clustering  of  Hadamard  frame  110  (the 
root  node)  produces  the  three  nodes  depicted  in  row  two  of  the  figure.  These  nodes  are  still 
depicting  frame  110,  as  at  the  root.  Notice  that  the  target  pixels  in  the  middle  cluster  do 
not  separate  from  non-target  pixels.  However,  Hadamard  frame  151  (the  next  node  in  the 
tree)  for  these  same  pixels  does  provide  significant  separability. 

5.  Conclusions  and  Discussion 

We  have  successfully  applied  ISPDT  to  DMA  hyperspectral  imagery  for  detec¬ 
tion/localization,  demonstrating  the  potential  of  an  integrated  sensing/processing  suite  con¬ 
sisting  of  a  dynamically  programmable  DMA  sensor  and  ISPDT  processing. 

This  example  demonstrates  detection  and  localization.  When  multiple  target  types  are 
possible,  a  subsequent  classification  step  —  possibly  involving  additional  Hadamard  frames, 
is  employed. 

If,  as  is  likely  in  practice,  detection  is  not  so  “perfect”  and  some  few  off-target  pixels  are 
identified  as  falling  into  the  “target  leaf”  (but  still  many  on-target  pixels  are  so  identified) 
then  post-processing  under  some  spatial  dependence  scheme,  such  as  maximum  a  posteriori 
(MAP)  spatial  filtering,  can  be  used  to  perform  the  ultimate  detection/localization. 

Finally,  while  it  may  seem  (and  is  indeed  the  case)  that  processing  more  elaborate  than 
our  individual  pixel-based  approach  (such  as  the  use  of  an  edge  detector)  would  make  the 
detection  and  localization  task  trivial,  the  extremely  time-sensitive  nature  of  the  pattern 
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recognition  applications  envisioned  here  preclude  elaborate  processing  schemes. 


Acknowledgments 

Work  supported  in  part  by  the  Applied  and  Computational  Mathematics  Program  (ACMP) 
of  the  Defense  Advanced  Research  Projects  Agency  (DARPA). 

References 

1.  C.E.  Priebe,  D.J.  Marchette,  and  D.M.  Healy,  “Integrated  Sensing  and  Processing  De¬ 
cision  Trees,”  IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence,  26  (6),  699-708 
(2004). 

2.  R.A.  DeVerse,  R.R.  Coifman,  A.C.  Coppi,  W.G.  Fateley,  F.  Geshwind,  R.M.  Hammaker, 
S.  Valenti,  F.J.  Warner,  and  G.L.  Davis,  “Application  of  spatial  light  modulators  for  new 
modalities  in  spectrometry  and  imaging,”  Proc.  SPIE  4959,  12-22  (2003). 

3.  M.  Harwit  and  N.J.A.  Sloane,  Hadamard  Transform  Optics  (Academic  Press,  New  York, 
1979). 

4.  R.A.  DeVerse,  R.M.  Hammaker,  and  W.G.  Fately,  “Realization  of  the  Hadamard  mul¬ 
tiplex  Advantage  Using  a  Programmable  Optical  Mask  in  a  Dispersive  Flat-Field  Near- 
Infrared  Spectrometer,”  Applied  Spectroscopy,  54  (12),  1751-1758  (2000). 

5.  A.  Wuttig,  R.  Riesenberg,  “Sensitive  Hadamard  Transform  Imaging  Spectrometer  with 
a  simple  MEMS,”  Proc.  SPIE  4881,  167-178  (2003). 

6.  L.  Hubbert,  and  P.  Arabie,  “Comparing  partitions,”  Journal  of  Classification,  2  193-218 
(1985). 

7.  W.M.  Rand,  “Objective  criteria  for  the  evaluation  of  clustering  methods,”  Journal  of 
the  American  Statistical  Association,  66  846-850  (1971). 

8.  C.  Fraley  and  A.E.  Raftery,  “Model-based  clustering,  discriminant  analysis,  and  density 
estimation,”  Journal  of  the  American  Statistical  Association,  97  611-631  (2002). 


7 


Fig.  1.  Hadamard  frame  110  (full  256x256  image)  for  / 2.  T2FP110-256x256.eps 
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Fig.  2.  Hadamard  frame  110  (64x210  swath)  for  1-2,  with  target  box  used  for 
training.  T2FPll0-64x210.eps 
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Fig.  3.  ISPDT  constructed  on  I2  (training).  The  left-most  node  at  level  three 
of  the  tree  is  the  “target  leaf”.  psstreeT2.eps 


10 


Fig.  4.  ISPDT  detection/localization  results  for  Iq  (testing).  No  vehicle  is 
present  at  time  t  =  0,  and  no  pixels  falling  into  the  “target  leaf”  implies 
no  detection,  as  desired.  psstreeT0.eps 
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Fig.  5.  ISPDT  detection/loealization  results  for  Ix  (testing).  There  is  a  vehicle 
present  at  time  t  =  1,  and  the  existence  of  pixels  falling  into  the  “target  leaf” 
implies  detection,  as  desired.  The  spatial  location  of  these  pixels  indicates  that 
the  detection  is  indeed  “on  target”.  psstreeTl.eps 
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Fig.  6.  ISPDT  detection/localization  results  for  I3  (testing).  There  is  a  vehicle 
present  at  time  t  =  3,  and  the  existence  of  pixels  falling  into  the  “target  leaf” 
implies  detection,  as  desired.  The  spatial  location  of  these  pixels  indicates  that 
the  detection  is  indeed  “on  target” .  psstreeT3.eps 
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Fig.  7.  ISPDT  detection/localization  results  for  /4  (testing).  There  is  a  vehicle 
present  at  time  t  =  4,  and  the  existence  of  pixels  falling  into  the  “target  leaf” 
implies  detection,  as  desired.  The  spatial  location  of  these  pixels  indicates  that 
the  detection  is  indeed  “on  target”.  psstreeT4.eps 
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Fig.  8.  Illustration  of  a  more  elaborate  ISPDT,  requiring  non-trivial  condition¬ 
ing/adaptation,  for  Iz .  psstreeT3-2.eps 
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Fig.  9.  Scatter  plot  tree  for  the  illustrative  ISPDT  depicted  in  Figure  8.  See 


text  for  description.  pssscatterT3.eps 

16 


